Simulation system implementing real-time machine data

ABSTRACT

A simulation and control system for a machine is disclosed. The simulation and control system may have a user interface configured to display a simulated environment. The machine simulation and control system may also have a controller in communication with the user interface and a remotely located machine. The controller may be configured to receive from the machine real-time information related to operation of the machine at a worksite. The controller may also be configured to simulate the worksite, operation of the machine, and movement of a machine tool based on the received information. The controller may further be configured to provide to the user interface the simulated worksite, operation, and movement in the simulated environment.

TECHNICAL FIELD

This disclosure relates generally to a simulation system and, moreparticularly, to a system that uses real-time performance data toremotely simulate operation of a machine at a worksite.

BACKGROUND

Machines such as, for example, excavators, loaders, dozers, motorgraders, haul trucks, and other types of heavy equipment are used toperform a variety of tasks. During the performance of these tasks, themachines may operate in situations that are hazardous to an operator,under extreme environmental conditions uncomfortable for the operator,or at work locations remote from civilization. Because of these factors,the completion of some tasks by an onboard operator can be dangerous,expensive, labor intensive, time consuming, and inefficient.

One solution to this problem may include remotely controlling themachines. Specifically, an offboard operator located remotely from themachine, if provided with a visual representation of the machine and thework environment, could control operation of the machine from a moresuitable location. This strategy has been implemented in the past andgenerally included providing the visual representation of the machineand work environment by way of live video feed broadcast from theworksite to the operator. The operator then was able to provide, via agraphical user interface, operational instructions that weresubsequently sent to the machine for control thereof

Although this strategy of remotely controlling the machines may havebeen successful in some situations, its use was limited and costly.Specifically, the visual representation of the machine and environmentwas typically limited to the number of cameras mounted to the machineand the view angles provided by those cameras. To improve visibility orprovide different view angles, additional cameras had to be installed onthe machine. Because the number of cameras on the machine relatesdirectly to cost and, because the harsh environment of the worksitereduced the component life of the cameras, the initial and operatingcosts of the system were significant. In addition, wireless video feedin real-time requires large bandwidth, thereby further increasing theoperating cost of the system.

An attempt at addressing the problems of high system cost and largebandwidth is described in U.S. Pat. No. 6,739,078 (the '078 patent)issued to Morley et al. on May 25, 2004. Specifically, the '078 patentdescribes a system utilized to remotely control construction equipmentsuch as a backhoe at an isolated location via a data network, in which auser provides movement instructions via a graphical user interface (GUI)at a user PC. The GUI displays a side view and a top view visualrepresentation of the movable elements of the backhoe (e.g., a boom, astick, and a bucket). The visual representation is generated in responseto movements of the boom, stick, and bucket of the backhoe, which aremeasured onboard the backhoe and transmitted to the user PC via radiofrequencies. In this manner, an operator may remotely control hydraulicactuators onboard the backhoe to move the boom, stick, and bucket and,at the same time, view the resulting motions at a distant location in acost effective manner.

Although the system of the '078 patent may provide a lower cost, morerobust way to remotely view and control motions of constructionequipment, its use may still be limited. In particular, because thesystem of the '078 patent provides a visual representation of only theboom, stick, and bucket, the operator may be unable to properly controlengagement of the backhoe with its surrounding environment. Inparticular, without a representation of the worksite or an excavationsurface at the work site, it may be very difficult, if not impossible,to adequately engage the bucket with the excavation surface. Inaddition, with the minimal visual representation described above, theoperator may be unable to remotely move or orient the backhoe itself, orperform other necessary machine tasks.

The system of the present disclosure is directed towards overcoming oneor more of the problems as set forth above.

SUMMARY OF THE INVENTION

In accordance with one aspect, the present disclosure is directed towarda machine simulation and control system. The machine simulation andcontrol system may include a user interface configured to display asimulated environment. The machine simulation and control system mayalso include a controller in communication with the user interface and aremotely located machine. The controller may be configured to receivefrom the machine real-time information related to operation of themachine at a worksite. The controller may also be configured to simulatethe worksite, operation of the machine, and movement of a machine toolbased on the received information. The controller may further beconfigured to provide to the user interface the simulated worksite,operation, and movement in the simulated environment.

According to another aspect, the present disclosure is directed toward amethod of remotely controlling a machine. The method may includemonitoring machine operation and simulating in real-time a worksite,machine movement within the worksite, and machine tool movement withinthe worksite based on the monitored machine operation. The method mayfurther include receiving machine control instructions at a locationremote from the machine, and affecting operation of the machine inresponse to the received instructions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial illustration of an exemplary disclosed machinetraveling about a worksite;

FIG. 2 is a schematic and diagrammatic illustration of an exemplarydisclosed simulation and control system for use with the machine of FIG.1;

FIG. 3 is a pictorial illustration of an exemplary disclosed graphicaluser interface for use with the system of FIG. 2; and

FIG. 4 is another pictorial illustration of the exemplary disclosedgraphical user interface for use with the system of FIG. 2.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary machine 10 performing a predeterminedfunction at a worksite 12. Machine 10 may embody a stationary or mobilemachine, with the predetermined function being associated with aparticular industry such as mining, construction, farming,transportation, power generation, or any other industry known in theart. For example, machine 10 may be an earth moving machine such as theexcavator depicted in FIG. 1, in which the predetermined functionincludes the removal of earthen material from worksite 12 that altersthe geography of worksite 12 to an architecturally desired form. Machine10 may alternatively embody a different earth moving machine such as amotor grader or a wheel loader, or a non-earth moving machine such as apassenger vehicle, a stationary generator set, or a pumping mechanism.Machine 10 may embody any suitable operation-performing machine.

As illustrated in FIG. 2, machine 10 may include a simulation system 14having multiple components that interact to monitor the operation ofmachine 10 and perform analysis in response thereto. In particular,machine 10 may include a data module 16 in communication with acontroller 18. It is contemplated that data module 16 and controller 18may be integrated in a single unit, if desired. It is furthercontemplated that simulation system 14 may include additional ordifferent components than those illustrated in FIG. 2.

Data module 16 may include a plurality of sensing devices 16 a-fdistributed throughout machine 10 to gather real-time data from variouscomponents and systems of machine 10. Sensing devices 16 a-f may beassociated with, for example, a work tool 20, a power source 22, atransmission device 24, one or more actuator devices 26, a positionlocating device 28, driven and/or steerable traction devices 30, atorque converter (not shown), a fluid supply (not shown), operator inputdevices (not shown), and/or other systems and components of machine 10.These sensing devices 16 a-e may automatically gather real-time datafrom machine 10, such as manipulation of tool 20, operation of powersource 22, and/or machine travel characteristics (e.g., speed, torque,track slip rate, etc.); orientation and position of machine 10; fluidpressure, flow rate, temperature, contamination level, and/or viscosity;electric current and/or voltage levels; fluid (i.e., fuel, oil, water,etc.) consumption rates; loading levels (e.g., payload value, percent ofmaximum allowable payload limit, payload history, payload distribution,etc.); transmission output ratio; cycle time; idle time, grade; recentlyperformed maintenance and/or repair operations; and other such pieces ofinformation. Additional information may be generated or maintained bymachine data module 16 such as the date, time of day, and operatorinformation. The gathered data may be indexed relative to the time, day,date, operator, or other pieces of information, and communicated tocontroller 18 to trend the various operational aspects of machine 10, ifdesired.

For example, a first sensing device 16 a may be associated with positionlocating device 28 to gather real-time machine position, orientation (adirection machine 10 is facing), and/or ground speed information. In oneaspect, locating device 28 may include a global positioning system (GPS)comprising one or more GPS antennae disposed at one or more locations onmachine 10 (e.g. on work tool 20, the front and/or rear of machine 10,etc.). The GPS antenna may receive one or more signals from one or moresatellites. Based on the trajectories of the one or more signals,locating device 28 may determine a global position and orientation ofmachine 10 in coordinates with respect to worksite 12. Further, byrepeatedly sampling machine positions, locating device 28 may determinea real-time machine ground speed based on distances between samples andtime indices associated therewith, or a time between samples.Alternatively, machine position, orientation, and/or ground speedinformation may similarly be determined in site coordinates with respectto a ground-based location. It is to be appreciated that otherpositioning methods known in the art may be used alternatively oradditionally.

In a further aspect, sensing device 16 a may gather pitch and roll datain order to determine a real-time inclination of machine 10 with respectto the surface of worksite 12. For example, if locating device 28includes three (or more) GPS antennae receivers disposed about machine10 as discussed above, pitch and roll angles of machine 10 may bedetermined by comparing an orientation of a surface defined by therespective positions of the three (or more) receivers relative to agravity vector and/or horizontal ground. Alternatively, sensing device16 a may be associated with conventional pitch and role inclinationelectronics disposed on machine 10. The electronics may include, forexample, electrodes disposed within a glass vial and submerged in anelectrically conductive fluid, such that as machine inclination changes,submersion depths of the electrodes also change, and electricalresistances of paths between electrodes may change accordingly. As such,the pitch and roll of machine 10 may be defined in terms of the measuredresistances. It is to be appreciated that other pitch and roll and/orinclination sensors known in the art may be used alternatively oradditionally.

A second sensing device 16 b, for example, may be associated withtraction devices 30 to gather real-time speed and/or velocity datathereof. For example, sensing device 16 b may be able to determine areal-time rotational speed of traction devices 30. It is to beappreciated that a track slip rate of traction devices 30 (i.e., a rateat which traction devices 30 are spinning in place) may be indicated bya detected difference between machine ground speed, as discussed above,and traction device speed. Alternatively, track slip rate may beindicated by a sudden increase in the speed of one or more of tractiondevices 30 detected by sensing device 16 b.

In another aspect, sensing device 16 b may gather real-time steeringcommand information. For example, in a case where traction devices 30comprise driven, non-steerable belts or tracks, a measured differencebetween rotational speeds thereof may indicate a corresponding turningrate and direction negotiated by machine 10. In another aspect, whereintraction devices 30 comprise steerable wheels, or the like, sensingdevice 16 b may simply measure a current steering angle thereof.

A third sensing device 16 c, for example, may be associated withtransmission device 24 to gather real-time data concerning a presenttransmission output (e.g., gear) utilized by machine 10. Additionally,sensing device 16 c may gather real-time data concerning a torque outputof transmission device 24. A fourth sensing device 16 d may beassociated with power source 22 in order to gather information regardinga speed output (RPM) and/or a torque output thereof.

A fifth sensing device 16 e may be associated with hydraulic devices 26to gather real-time data related to positioning of a linkage system 32and/or tool 20. For example, actuator devices 26 may comprise hydrauliccylinders extendible throughout a range between a minimum length and amaximum length. In conjunction with known kinematics and geometry oflinkage system 32 and/or tool 20, a three-dimensional position andorientation thereof, in site coordinates, may be determined based onsensed extension lengths of hydraulic devices 26.

A sixth sensing device 16 f, for example, may be associated with tool 20to gather real-time data concerning a load applied thereto. The load maybe represented as a force, weight, volume, and/or mass of materialengaged or supported by tool 20. Additionally, the load may bedetermined as a percentage of a maximum capacity load (i.e., a fullload) that may be engaged or supported by tool 20. The maximum capacityload may be based on known specifications of linkage system 32, tool 20,and/or other components of machine 10. For example, if tool 20 comprisesa bucket, device 16 f may include a scale mechanism that may directlydetermine a force, weight, volume, and/or mass of the material therein.Alternatively, device 16 f may comprise one or more optical sensorsdisposed about an inner engagement surface of tool 20 to sense acapacity to which tool 20 is filled with the material. Based on knownspecifications, a volume of material engaged by tool 20 may bedetermined. In another aspect, sensing device 16 f may measure a forceexerted by hydraulic devices 26 to maintain tool 20 in a desiredposition. As such, the measured force, in conjunction with known torquerelationships between linkage system 32 and tool 20, and otherspecifications of machine 10, may allow determination of the force,weight, mass, volume, and/or percent capacity of the load. It is to beappreciated that other methods of load sensing known in the art may beused alternatively or additionally.

Controller 18 may be in communication with data module 16 and includeany means for monitoring, recording, storing, indexing, processing,and/or communicating the real-time data concerning operational aspectsof machine 10 described above. These means may include components suchas, for example, a memory, one or more data storage devices, a centralprocessing unit, or any other components that may be used to run anapplication. Furthermore, although aspects of the present disclosure maybe described generally as being stored in memory, one skilled in the artwill appreciate that these aspects may be stored on or read fromdifferent types of computer program products or computer-readable mediasuch as computer chips and secondary storage devices, including harddisks, floppy disks, flash drives, optical media, CD-ROM, or other formsof RAM or ROM.

Controller 18 may further include a means for communicating with anoffboard, remotely-located user interface 34. For example, controller 18may include hardware and/or software that enables transmitting andreceiving of the data through a direct data link (not shown) or awireless communication link (not shown). The wireless communications mayinclude satellite, cellular, infrared, radio, microwave, or any othertype of wireless electromagnetic communications that enable controller18 to exchange information. It is contemplated that a separate modulemay alternatively be included within simulation system 14 to facilitatethe communication of data between controller 18 and user interface 34,if desired. In one aspect, controller 18 may communicate the data to abase station 36 equipped to relay the communications to user interface34. Other simulation-capable machines associated with worksite 12 mayalso similarly communicate data to base station 36. Subsequently, thedata may be communicated to an intermediary, such as a server (notshow), which may appropriately package and transmit the received data touser interface 34 for simulation.

User interface 34 may represent one or more receiving, computing, and/ordisplay systems of a business entity associated with machine 10, such asa manufacturer, dealer, retailer, owner, service provider, client, orany other entity that generates, maintains, sends, and/or receivesinformation associated with machine 10. The one or more computingsystems may embody, for example, a machine simulator, a mainframe, awork station, a laptop, a personal digital assistant, and othercomputing systems known in the art. Interface 34 may include componentssuch as, for example, a memory, one or more data storage devices, acontroller 38 (CPU), or any other components that may be used to run anapplication. In one aspect, interface 34 may include a firewall and/orrequire user authentication, such as a username and password, in orderto prevent access thereto by unauthorized entities.

User interface 34 may be operatively coupled to communicate with aworksite terrain map 42. Terrain map 42 may include work surface datadefining ground elevation, earthen material composition and/orconsistency at a plurality of locations at worksite 12 defined in sitecoordinates. Additionally, terrain map 42 may include the location,size, shape, composition, and/or consistency of above- or below-groundobstacles at, or in the proximity of worksite 12, such as, for example,roads, utility lines, storage tanks, buildings, property boundaries,trees, bodies of water, and/or other obstacles. In one aspect, terrainmap 42 may be a predetermined schematic CAD rendering or the like. Inanother aspect, terrain map 42 may be generated by geographic sensingequipment (not shown), such as for example, a ground-penetrating radarsystems (GPR) associated with machine 10 and/or worksite 12, and/orsatellite imagery equipment known in the art. It is to be appreciatedthat terrain map 42 may include work surface data concerning a pluralityof predetermined worksites that may or may not be related to worksite12.

Terrain map 42 may be stored within the memory, data storage devices,and/or central processing unit of controller 18 and communicated to userinterface 34 in conjunction with the gathered real-time information.Alternatively, terrain map 42 may be stored within the memory, datastorage devices, and/or controller 38 of user interface 34. In anotheraspect, terrain map 42 may be stored in a separate location andcommunicated to user interface 34. Further, terrain map 42 may comprisea database compatible with the real-time information gathered by datamodule 16. In one aspect, controller 18 may update terrain map 42 basedon the received real-time data to reflect changes affected upon worksite12 as a result of machine position during travel, and/or tool movementand loading sensed during excavation maneuvers. This feature will bediscussed further in the next section to illustrate use of the disclosedsimulation system 14.

User interface 34 may further include one or more monitors 44 configuredto actively and responsively display a simulated environment of machine10 on worksite 12, as well as parameters indicating machine performanceand functionality, in response to the received real-time data andterrain map 42. Monitor 44 may include, for example, a liquid crystaldisplay (LCD), a CRT, a PDA, a plasma display, a touch-screen, aportable hand-held device, or any such display device known in the art.In one aspect, monitors 44 may comprise a full 360-degree displayencompassing the operator for augmented, realistic display of thesimulated worksite 12.

As illustrated in FIGS. 3 and 4, user interface 34 may generate anddisplay one or more selectable 3-D viewpoint perspectives 48 of machine10 and worksite 12 on monitor 44 in response to the received real-timedata and based on terrain map 42 for remote control of machine 10. Assuch, the components of interface 34 may be tailored to render 3-Denvironments in conjunction with the machine control application. Forexample, one viewpoint 48 a may correspond with a high-level,third-person view of machine 10, as it is controlled and moved aboutworksite 12, resembling the image of FIG. 3. From this viewpoint, anoperator may discern and control, among other things, an optimal travelroute and/or approach to an excavating location, work pile, or otherpoint of interest on worksite 12.

A second viewpoint 48 b may correspond with a close look at work toolmovement from inside or outside of operator station 46, and may resemblethe image of FIG. 4. From this viewpoint, an operator may discern and/orcontrol, among other things, work tool and linkage system movement andloading during excavation passes, and the results of excavation from agiven machine position. Additionally, details such as the contour andlayout of work surface terrain proximate the machine position may beaccurately depicted in second viewpoint 48 b. Further, nearby obstaclesincluded in terrain map 42, such as buildings, roads, trees, and/orproperty boundaries, etc., may also be accurately depicted in secondview point 48 b. As such, the operator may easily determine if a currentterrain of worksite 12, as indicated by terrain map 42, is compatiblewith a desired terrain or if additional excavation passes are required,or if a desired machine maneuver may be obstructed by the surroundingobstacles and/or work surface terrain.

In a further aspect, views may be selectable from any desired referencepoint, since simulation may not be limited to a finite number ofstationary cameras, but generated according to the terrain map 42 andthe received real-time data. However, it is to be appreciated that somevideo may be gathered by one or more cameras mounted on machine 10 andcommunicated to user interface 34 in addition to the gathered real-timedata. The video feed may be utilized and enhanced with simulation basedon the received data and terrain map 42, and provided to the operator byway of monitor 44.

As illustrated in FIGS. 3 and 4, offboard system 34 may provide to aremote operator of machine 10 an onboard visual indication of theperformance of machine 10 based on the received real-time information.For example, an information panel 50 may be included within the displayarea on monitor 44. Information panel 50 may include a plurality ofindicators 50 a-i associated with respective parameter values derivedfrom the received real-time information.

For example, panel 50 may include a machine ground speed indicator 50 ato show the present ground speed of machine (mph or km/h), an enginespeed indicator 50 b to show the present engine rotational speed (RPM),a fuel level indicator 50 c, and/or a transmission output ratio (gear)indicator 50 d. Further, panel 50 may include slip indicator 50 e toidentify a rate at which traction devices 30 that may be slipping. Forexample, slip indicator 50 may show that the left track is slipping at arate of 0.2 mph. Panel 50 may also include a machine roll and pitchindicator 50 f to provide the operator with present inclination anglesof machine with respect to horizontal ground (e.g., 20-degree pitch and12-degree roll). Additionally, panel 50 may include a loading indicator50 g to show a capacity to which tool 20 is filled (e.g., 25%), and/or asteering command indicator 50 h to show a present steering angle oftraction devices 30 (e.g., 22-degrees left). Panel 50 may include otherindicators, such as, for example, a machine positioning indicator 50 ishowing a vertical overhead view of the position of machine relative toworksite 12 (e.g., a machine icon positioned on a map of worksite 12).Alternatively or additionally, machine position indicator 50 i mayindicate present latitude and longitude, and/or other coordinatesrepresenting a current position of machine 10 with respect to worksite12. It is to be appreciated that any other parameter values of interestmay be selectively provided in panel 50 based on the received real-timedata in order to provide an augmented reality for the machine operator.

Referring back to FIG. 2, user interface 34 may include an input device40 for remotely initiating operator command signals that controloperation of machine 10 at worksite 12. The command signals may becommunicated from user interface 34 to controller 18. For example,interface 34 may include a machine control application to receive theoperator command signals and appropriately package them for transmissionto controller 18. As such, controller 18 may generate machine commandsignals to control the various operational aspects of machine 10 inresponse to the received operator command signals. For example,controller 18 may vary electrical signals, hydraulic pressure, fluidflow rates, fluid consumption levels, etc., in order to change enginespeed, ground speed, transmission output ratio, steering angle, tool 20and/or linkage system 32 positioning in accordance with the receivedoperator commands.

In one aspect, input device 40 may resemble the operator interfaceincluded on machine 10. For example, input device 40 may include anarrangement of joysticks, wheels, levers, pedals, switches, and/orbuttons similar (or identical) to that of machine 10. As such, operatormanipulation of input device 40 may have the same effect on machine 10as corresponding manipulation of the operator interface within machine10. Input device 40 may be generic, and used for remote control of manydifferent types of simulation-capable machines 10. Alternatively, device40 may be customized for a specific type of machine (e.g., a 416EBackhoe Loader, or a 365C Hydraulic Excavator, manufactured byCaterpillar Inc., etc.), and include control features unique to themachine type. However, it is to be appreciated that device 40 may simplyembody one or more conventional computer interface devices, such as, forexample, a keyboard, touchpad, mouse, or any other interface devicesknown in the art.

Operation of the disclosed simulation system 14 will be discussedfurther the following section.

INDUSTRIAL APPLICABILITY

The disclosed simulation system may be applicable to any machine whereefficient control thereof from a remote location is desirable, and anaugmented, simulated operational environment may provide certainadvantages over live video feed. In particular, the disclosed simulationsystem may provide an augmented display based on real-time datameasurements that include multiple simulated views of the machine, theworksite, and various operational parameter values, such that anoperator may comfortably and effectively control the machine. Operationof simulation system 14 will now be described.

In one aspect, an operator may log into user interface 34 by entering ausername and password, and initiate the remote machine controlapplication. The operator may then be prompted to select a desiredworksite. For example, controller 38 of user interface 34 may retrieve aplurality of available worksites from terrain map 42 and display them onmonitor 44. The operator may then use input device 40 to navigatethrough and select a desired worksite 12 from among the plurality.

Subsequently, controller 38 may receive terrain information aboutselected worksite 12 from terrain map 42 and generate a simulated 3-Denvironment of worksite 12. As discussed above, the environment mayinclude a surface of the terrain, obstacles thereon, and/or plan linesassociated with the worksite 12. Controller 38 may then receive, fromterrain map 42, or controllers on individual machines, positioninformation regarding a plurality of available simulation-capablemachines associated with worksite 12. As such, controller 38 may displayeach available machine on monitor 44, and prompt the operator to selecta desired machine 10 for operation. It is to be appreciated that eachoperator may be authorized to access different worksites and/or machinesfor a variety of reasons. As such, the worksites and/or machinesavailable to the operator may be a function of the operator's usernameand/or password or an operator profile associated therewith.

Once a worksite 12 and a machine 10 have been properly accessed,controller may begin receiving the streaming real-time data gathered bythe particular machine 10. Additionally, controller 38 may beginreceiving streaming real-time data from other simulation-capablemachines associated with worksite 12. In one aspect, in order toconserve bandwidth and/or processing power, the real-time data receivedfrom other machines may be limited to certain parameters of interest,such as, for example, position and/or travel speed thereof. However, itis to be appreciated that controller 38 may receive any desiredproportion of gathered real-time data from any number ofsimulation-capable machines associated with worksite 12. For example,controller 38 may receive real-time data concerning tool movement and/orloading of other machines for augmented simulation of the worksiteenvironment.

Upon receiving the streaming real-time data, controller 38 may activelypopulate the 3-D environment with the machines 10 associated withworksite 12, and display the populated environment to the operator onmonitor 44. Controller 38 may then prompt or otherwise allow theoperator to initiate a command by way of input device 40 to startmachine 10. Subsequently, machine 10 may be controlled and moved aboutworksite 12 by way of input device 40, as discussed above.

Further, controller 38 may allow the operator to select, by way of inputdevice 40, a desired viewpoint from one or more available commonly-usedviewpoints, such as, for example, one of the viewpoints 48 discussedabove in connection with FIGS. 3 and 4, or a viewpoint from theperspective of work tool 20. Alternatively or additionally, controller38 may provide a mode allowing the operator to define a viewpoint fromany desired perspective by way of input device 40. For example, theoperator may be able to select a first-person view of worksite 12 fromthe interior and/or exterior of machine, a third-person view of machine10 and worksite 12 that follows machine 10 during navigation, and/or aview of machine 10 and/or worksite 12 from a desired fixed location.Additionally, controller 38 may allow the operator to adjust a viewangle and/or zoom level associated with each of these perspectives. Itis to be appreciated that the operator may change the selected viewpointduring machine operation, if desired.

In one aspect, the operator may select the high-level, third-personperson perspective 48 a of FIG. 3 and navigate machine 10 to a point ofinterest on worksite 12, such as, for example, a pre-determinedexcavating location delineated by plan lines 52, by manipulating inputdevice 40, as discussed above. Accordingly, the received real-timeinformation may responsively indicate machine navigation, and controller38 may actively update the view perspective 48 a and/or informationpanel 50 in response thereto. In other words, as machine 10 moves aboutthe worksite 12, the view perspective of the simulated 3-D environmentprovided on monitor 44 may change in accordance with the real-timemeasured machine ground speed, engine speed, fuel level, pitch and roll,transmission output ratio, tool position, etc.

Controller 38 may also provide certain augmented display features inorder to improve operator control of machine 10. For example, if, duringnavigation, the received real-time data indicates that a traction device30 is slipping, slip indicator 50 e may indicate the appropriatetraction device and the rate at which it is slipping (e.g., left at 0.2mph). Additionally, if the operator is utilizing a high-level,third-person perspective 48 a where the traction devices 30 are visiblein the simulated environment (FIG. 4), controller 38 may indicate aslipping traction device by coloring, flashing, or otherwise visuallydistinguishing the traction device from the background environment.Alternatively or additionally, controller 38 may distinguish a slippingtraction device simply by showing the traction device rotating orotherwise moving more quickly than the machine ground speed.

Upon reaching a point of interest on worksite 12, the operator mayselect a viewpoint 48 b corresponding with a close look at work tool 20from inside operator station 46 in order to facilitate excavation withinplan lines 52. As the operator manipulates input device 40 in order tocontrol work tool movement, controller 38 may actively simulate anddisplay tool 20 and linkage system 32 movement in response to thereceived real-time data. For example, as the components of linkagesystem 32 (e.g., boom, stick, and bucket 20) are tilted downward orotherwise moved toward a work surface, sensors 16 e may providereal-time position signals to module 16 for communication to userinterface 34. Upon receiving the communication, controller 38 may showlinkage system 32 (e.g., boom, stick and bucket 20) moving at themeasured velocity to the measured position based on the real-timeposition signals.

Additionally, as the operator manipulates input device 40 in order tomake excavation passes with work tool 20, and earthen material isremoved from worksite 12, controller may actively update the simulatedenvironment terrain shown in view perspective 48 b. For example, as tool20 engages and removes material from a given point on the work surface,sensor 16 f may provide a real-time loading signal to module 18 forcommunication to user interface 34. Upon receiving the communication,and in conjunction with the linkage system 32 and tool 20 positioningcommunication discussed above, controller 38 may determine an amount ofmaterial removed from the work surface, and the location from which itwas removed, upon completion of each excavation pass. As such,controller 38 may responsively update view perspective 48 and/or terrainmap 42 during completion of an excavation pass to reflect geographicalchanges made to worksite 12. Further, if other simulation-capablemachines are performing excavation on worksite 12, controller 38 maysimilarly update the view perspective 48 b and/or terrain map 42 inresponse to received communications of real-time information concerningmachine location, linkage system and work tool positioning, movement,and loading thereof

Although the forgoing disclosure relates to generating a 3-D simulationof a worksite environment, it is to be appreciated that supplementalvideo feed may be used in conjunction therewith. For example, machine 10may be equipped with one or more cameras, and real-time video signalsmay be communicated to user interface 34 in addition to the real-timegathered data. As such, controller 38 may provide a live video feed ofworksite 12 to the operator by way of monitor 44, which may, in turn, beaugmented with simulation based on the real-time gathered data, asdiscussed above. The proportion of live video to augmented simulationmay be selectable by the operator and determined, in part, based on adesired simulation quality and the availability of necessary systemresources, such as, for example, processing power and bandwidth. Forexample, for a given amount of available resources, the operator may beable to select a certain degree of live video feed (e.g., three cameraviews) in addition to certain simulated parameters (e.g., pitch androll, track slip, and machine ground speed). However, it is to beappreciated that any desired proportion or combination of live videofeed and/or augmented simulation may be used, within available resourcelimitations, as desired.

Because controller 38 may generate a 3-D environment in response toreceived real-time data associated with various operational parametersof machine 10, remote control of machine 10 may be facilitated without,or with minimized use of live video feed, which requires largebandwidth. In particular, the real-time data may be communicated to userinterface 34 by way of radio signals or other low-bandwidth carriers,where it may be used by controller 38 to render a simulated 3-Denvironment of worksite 12. Moreover, since controller 38 may processthe received data in order to provide different view perspectives ofmachine 10 with respect to worksite 12, visibility may not be limited tothe number of cameras provided on the machine or respective fields ofview associated therewith.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the method and system of thepresent disclosure. Other embodiments of the method and system will beapparent to those skilled in the art from consideration of thespecification and practice of the method and system disclosed herein. Itis intended that the specification and examples be considered asexemplary only, with a true scope of the disclosure being indicated bythe following claims and their equivalents.

1-20. (canceled)
 21. A system for remotely controlling a machine on asite, the system comprising: an operator interface; a communicationdevice; and a control unit configured to: identify one or more sites atwhich one or more machines are available for remote control; allow anoperator, via the operator interface, to select one of the sites atwhich one or more machines are available for remote control, and toselect one of the available machines at the selected site; receive, viathe communication device, real-time information related to operation ofthe selected machine at the selected site; simulate operation of theselected machine at the selected site based on the received real-timeinformation and on an electronic map associated with the selected site;provide the simulation via the operator interface; and receive, via theoperator interface, machine control commands from the operator; andcommunicate the received machine control commands to the selectedmachine via the communication device.
 22. The system of claim 21,wherein the control unit is further configured to: allow the operator,via the operator interface, to select a view reference perspective ofthe selected machine from one or more view references perspectives; andprovide the simulation according to the selected view referenceperspective of the selected machine.
 23. The system of claim 22, whereinthe view reference perspectives include a first-person view from theselected machine and a third-person view of the selected machine. 24.The system of claim 21, wherein the real-time information related tooperation of the selected machine includes positioning information, andthe control unit is further configured to indicate, in the simulation, apositioning of the machine at the selected site based on the positioninginformation.
 25. The system of claim 21, wherein the real-timeinformation related to operation of the selected machine includes speedinformation, and the control unit is further configured to indicate, inthe simulation, a speed of the selected machine based on the speedinformation.
 26. The system of claim 21, wherein the real-timeinformation related to operation of the selected machine includessteering information, and the control unit is further configured toindicate, in the simulation, a steering magnitude of the machine basedon the steering information.
 27. The system of claim 21, wherein thereal-time information related to operation of the selected machineincludes roll and pitch information, and the control unit is furtherconfigured to indicate, in the simulation, a roll and pitch of theselected machine based on the roll and pitch information.
 28. The systemof claim 21, further comprising data storage storing electronic mapsassociated with respective sites, wherein the control unit is configuredto access the electronic map associated with selected site from the datastorage.
 29. A computer-assisted method for remotely controlling amachine on a site, the method comprising: identify one or more sites oneat which one or more machines are available for remote control; allowingan operator, via an operator interface, to select one of the sites atwhich one or more machines are available for remote control, and toselect one of the available machines at the selected site; receivingreal-time information related to operation of the selected machine atthe selected site; simulating operation of the selected machine at theselected site based on the received real-time information and on anelectronic map associated with the selected site; providing thesimulation via the operator interface; and receiving, via the operatorinterface, machine control commands from the operator; and communicatingthe received machine control commands to the selected machine.
 30. Themethod of claim 29, further comprising: allowing the operator, via theoperator interface, to select a view reference perspective of theselected machine from one or more view references perspectives; andproviding the simulation according to the selected view referenceperspective of the selected machine.
 31. The method of claim 30, whereinthe view reference perspectives include a first-person view from theselected machine and a third-person view of the selected machine. 32.The method of claim 29, wherein the real-time information related tooperation of the selected machine includes positioning information, andthe method further includes indicating, in the simulation, a positioningof the machine at the selected site based on the positioninginformation.
 33. The method of claim 29, wherein the real-timeinformation related to operation of the selected machine includes speedinformation, and the method further includes indicating, in thesimulation, a speed of the selected machine based on the speedinformation.
 34. The method of claim 29, wherein the real-timeinformation related to operation of the selected machine includessteering information, and the method further includes indicating, in thesimulation, a steering magnitude of the machine based on the steeringinformation.
 35. The method of claim 29, wherein the real-timeinformation related to operation of the selected machine includes rolland pitch information, and the method further includes indicating, inthe simulation, a roll and pitch of the selected machine based on theroll and pitch information.
 36. A computer-assisted method for remotelycontrolling a machine on a site, the method comprising: identify one ormore machines at a site that are available for remote control; allowingan operator, via an operator interface, to select one of the availablemachines for remote control; receiving real-time information related tooperation of the selected machine; simulating operation of the selectedmachine at the site based on the received real-time information and onan electronic map associated with the site; providing the simulation viathe operator interface; and receiving, via the operator interface,machine control commands from the operator; and communicating thereceived machine control commands to the selected machine.
 37. Themethod of claim 36, wherein identifying one or more machines includes:identifying one or more sites one at which one or more machines areavailable for remote control; allowing an operator, via an operatorinterface, to select one of the sites at which one or more machines areavailable for remote control; and identifying one or more machines atthe selected site that are available for remote control.
 38. The methodof claim 36, further including: allowing the operator, via the operatorinterface, to select a view reference perspective of the selectedmachine from one or more view references perspectives; and providing thesimulation according to the selected view reference perspective of theselected machine.
 39. The method of claim 38, wherein the view referenceperspectives include a first-person view from the selected machine and athird-person view of the selected machine.
 40. The system of claim 36,wherein the real-time information related to operation of the selectedmachine includes positioning information, and the control unit isfurther configured to indicate, in the simulation, a positioning of themachine at the selected site based on the positioning information.